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In 1998, Angela Jerabek, a guidance counselor at a high school outside Minneapolis, 
grew frustrated watching too many ninth-graders fail courses year after year. Her prin- 
cipal encouraged her to come up with a new strategy to address the problem, knowing 
that failed classes too often lead students to drop out of high school.’ By the next school 
year, Jerabek had developed Building Assets, Reducing Risks (BARR) —a research- 
based program that “empowers educators to analyze real-time data and build upon indi- 


vidual strengths to support academic, social, and emotional success for every student.”” 


BARR’s focus on relationships and data was successful at Jerabek’s St. Louis Park High 
School: Course failure rates dropped from 47 percent to 28 percent in one year and 
stabilized at less than 20 percent in subsequent years. But despite these promising 
results, for more than a decade, the program did not expand beyond that one school. 
In 2010, when BARR won a grant from the first Investing in Innovation (i3) competi- 
tion, that changed as the program expanded to a suburban Los Angeles high school 


and two rural high schools in Maine.’ 


The American Recovery and Reinvestment Act (ARRA) of 2009 provided nearly $100 
billion to the U.S. Department of Education “to stimulate the economy in the short 
term and invest in education and other essential public services to ensure the long- 
term economic health of our nation.’* Nearly $49 billion from this stimulus was deliv- 
ered to governors to stabilize education funding and other state services during the 
recession, nearly $25 billion went to school districts through major formula programs, 


and $17 billion was dedicated to increasing Pell Grant funding. 


In addition, a small portion of ARRA’s funding—which still represented unprecedent- 
edly large dollar amounts—was reserved for competitive grant programs, including 
$650 million to create the i3 program, which awards grants to test, implement, and 
scale educational interventions. The amount of i3 funding that grantees received 

was divided into three levels, tied to the existing evidence for each program. Small 
“Development grants” went to new or weakly tested interventions, medium-sized 
“Validation grants” allowed interventions with some evidence to test their efficacy in 
new contexts or different populations, and large “Scale Up grants” supported wide- 


spread expansion of established interventions with strong prior evidence.* 


i3 and its renamed successor, the Education Innovation and Research (EIR) program, 
have invested approximately $1.7 billion through nearly 250 total grants to date.” 

But education research remains underfunded, especially in relation to other policy 
areas.* In fiscal year 2018, even if the EIR program's funding had been included, the 
Education Department’s spending on research and development (R&D) still would 
have been only slightly more than 1 percent of what the U.S. Department of Health 


and Human Services spent.’ 


In order to help identify and scale successful new ideas, the United States can and should 
invest more in education R&D, and increased support for EIR should be part of that 
effort. Below are some of the key lessons learned from the early years of the i3 program. 
Overall, the authors find that an emphasis on evidence and outcomes is possible and that 
focused research priorities can have advantages. But there are still clear challenges in scal- 
ing effective programs and building the demand for high-quality research. In the end, no 
one program can do everything, and R&D investments should be judged on the impacts 
of the innovations they support, not just on how many investments succeed. 


An emphasis on evidence and outcomes is possible 


The Department of Education’s focus on generating and using rigorous evidence began 
years before i3. The founding of the Institute of Education Sciences (IES) in 2002 
placed a focus on rigorous research—particularly randomized controlled trials—and 
the launch of the What Works Clearinghouse (WWC) allowed the IES to share results 
from studies that met high standards. This focus on evidence also was not a phe- 
nomenon that was unique to education as i3 launched, as evidence-based programs 
developed across government, including teen pregnancy prevention programs; home 
visiting programs that connected medical and child development professionals with 
first-time parents; and the Social Innovation Fund (SIF), which invests in the follow- 
ing “priority areas:” economic opportunity, healthy futures, and youth development.” 


Fueled by hundreds of millions of federal dollars and a requirement to get partially 
matching philanthropic grants, the i3 competition was designed to accelerate the 
Education Department’s evidence-based efforts with grants for school districts and 
nonprofits that funded high-quality evaluations of grantees’ programs as they expanded. 
These evaluations built the evidence base, even if they did not find positive effects.'! 


Beth Boulay, a principal associate at Abt Associates who helped lead evaluation techni- 
cal assistance to i3 grantees, told the authors that one of the most exciting outcomes of 
i3 is that more than 70 percent of the evaluations meet the rigorous WWC standards, 
creating a tremendous amount of high-quality research for the field.’” The initial i3 
competition attracted nearly 1,700 applications, and thousands more have been con- 
sidered in future rounds, even with much smaller regular appropriations after the stim- 
ulus, suggesting that the field has embraced this call for evidence-based solutions." 


While controversial, focused research priorities can have advantages 


The initial Investing in Innovation competition in 2010 offered significant latitude 

in the strategies and priorities that grantees could address. Even just among the four 
grantees in the scale-up grant tier, awards went to a range of topics: Teach For America 
addressed teacher preparation, the KIPP Foundation expanded its charter network 
and trained principals, the Success for All Foundation expanded a whole-school turn- 
around model, and the Reading Recovery Council of North America implemented 


reading interventions aimed at first-graders reading below grade level.'* 


However, when future rounds of the competition began to rely on smaller annual 
appropriations, rather than the one-time stimulus funding from the American 
Recovery and Reinvestment Act, the Department of Education developed more 
focused priorities, such as improving the effectiveness of principals and improving 
science, technology, engineering, and math education.'* Nadya Chinoy Dabby—the 
former assistant deputy secretary for the Office of Innovation and Improvement, 
which oversaw the program—told the authors that this strategy had multiple goals.'° 
Focused priorities allowed the Education Department to recruit peer reviewers with 
deep expertise in a particular area, create cohorts of grantees working on similar issues 
to learn from each other and provide more strategic technical assistance, and increase 
the odds that the field would learn something useful in a specific area. 


But the decision to focus the program’s awards on particular priorities proved contro- 
versial over concerns that the “administration started getting too heavy-handed with 
the [competition’s] priorities.”’” The use of priorities was somewhat reined in when i3 
was reauthorized and renamed Education Innovation and Research in 2015's Every 
Student Succeeds Act (ESSA), as the new program places “a greater emphasis on the 
research priorities of the field in setting priorities for grants.’'* But the most recent 
competition still retained an element of focus by inviting proposals for “Field-Initiated 
Innovations— General” and the more directed “Field-Initiated Innovations— Science, 
Technology, Engineering, and Math (STEM).””” A possible compromise could 

allow prioritization to be developed outside the Education Department’s politically 
appointed leadership, either by researchers at the Institute of Education Sciences or 
through an independent advisory body. 


It is difficult to identify effective solutions, 
and it may be even harder to scale them 


Education outcomes are affected by students, families, staff, schools, funding, policies, 
and an array of environmental factors that can all change. But even when programs 
such as i3 identify causal evidence that a program or intervention works, growth of 
those effective solutions across the more than 13,000 public school districts and nearly 
100,000 schools across the country is not a given.” 


In addition to associated growth challenges due to a lack of demand for evidence- 
backed practices, growing the supply of an effective intervention presents challenges. 
Especially for scale-up grants that already have significant evidence of effectiveness, 
the most important questions may pertain to which factors drove the success; whether 
they were consistent across different types of students, schools, and communities; 
how faithful the implementation was in new settings; and whether costs for future 
implementations can be reduced to make them less reliant on grant funding and more 


sustainable when grants run out. 


In an interview with the Social Innovation Research Center (SIRC) for the organi- 
zation’s report on i3, former Education Department Deputy Secretary Jim Shelton 
expressed a desire for more of an investment in implementation studies, rather than 
impact studies, in order to get a better understanding of what drives impacts and the 
reasons it does so.”' Dabby also stressed that evaluation requirements could better con- 
vey that a goal of evaluation should be to understand how to operate programs more 
efficiently by the end of the grant.” 


Building awareness of, and demand for, evidence is still a challenge 


According to the authors’ phone interview with Patrick Lester, Director of SIRC, 

i3’s grants had a major impact in the schools implementing effective interventions, 
but they are not yet having a broad impact on the field as a whole since diffusion has 
been a slow process.” i3 grantees were required to conduct activities to disseminate 
findings from their evaluations, but the dispersed network of grantees—ranging 
from relatively small school districts to large nonprofits—had varying capacity for 
outreach to reach key audiences. In interviews with the SIRC, some i3 grantees 
expressed a desire for the Education Department to play a bigger role in spreading 
the word about i3’s evidence, with one saying, “They need to disseminate to decision 


makers. As it is, each project does it on their own.” 


One of the Department of Education’s key mechanisms for disseminating research 
findings is the What Works Clearinghouse, but the initiative’s review process does 

not prioritize i3 studies, and grantees expressed frustration to the SIRC that there was 
a yearslong waitlist for those reviews. Besides, the findings that do make it onto the 
WWC are not guaranteed to reach educators. In a 2015 survey about research use, a 
majority of district leaders reported that they “rarely” or “never” searched for or found 
research using the WWC.”® 


Even when schools are aware of these evidence-based programs, there are often 
issues related to funding, internal capacity, and buy-in that can derail growth efforts, 
since schools may be interested in programs due to the grant funding that accompa- 
nies them rather than a commitment to implementing new practices. Among some 
scale-up grantees, there were higher-than-expected attrition rates even during the 
grant periods, as the availability of i3 funding may have attracted schools with insuf- 
ficient buy-in to the programs.” 
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One of the most promising avenues to increase demand for evidence-backed practices 
is the inclusion in the Every Student Succeeds Act of an evidence framework similar to 
i3 and the Education and Innovation Research program's requirements that now guide 
school improvement investments.” Lester notes that ESSA is encouraging adoption of 
more evidence-based practices and that some states are now developing lists of inter- 
ventions that meet their evidence standards.” 


No one program can do everything 


One of the frequent criticisms of the i3 program is that it did not really fund new or 
groundbreaking early-stage innovations.” And while that observation is somewhat 
true, Dabby emphasized in her interview that i3 should not be thought of as an entire 
innovation agenda on its own, as it was not designed to develop brand-new “stage zero” 
programs or interventions.” In her interview, Boulay similarly described it as a “floor,” 
or minimum requirement, for even the smallest tier of grants in i3 to have a “fully 


baked intervention” that was ready to be rigorously evaluated.*! 


In particular, i3 and EIR’s smallest grant tier presents challenges within the Education 
Department’s peer review processes, where grants are made solely based on reviewers’ 
assessments of what’s included in applications. Unlike private sector or philanthropic 
investments, reviewers do not interview applicants’ management teams to probe them 
on their program designs and plans for growth, nor do they conduct external due 
diligence.*” Changes to these processes could help the programs make more informed 


decisions on the newest and riskiest grants in the program. 


An analysis of the final evaluation reports from the original cohort of i3 grantees was 
published in 2018. It showed that only 18 percent of the grantees demonstrated a posi- 
tive impact on student academic outcomes, with many others having null results for a 
range of reasons, including sample size and other evaluation factors.** While one write- 
up labeled this a “dirty secret,’™ it actually represents a success rate that is somewhat 
better than those of other education interventions; for example, a 2013 Coalition for 
Evidence-Based Policy study found that only 12 percent of interventions yield positive 
effects.*° Moreover, the success rate of i3 grantees is comparable to those of clinical tri- 


als, which often range from 10 percent to 15 percent.*° 


Conclusion 


Nine years after receiving its first i3 grant, Building Assets, Reducing Risks is now the 
first program to have climbed through all three levels of the i3 competition.*’ A decade 
ago, this was a promising program in a single Minneapolis-area high school. Now, as a 
result of i3, BARR is on track to be used in 250 schools by 2021.** 
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Innovation is hard work. The i3 grants highlighted challenges building the supply of 
and demand for evidence-based practices, and the EIR program is still working toward 
striking a balance between focused priorities and leaving room for field-initiated innova- 
tions. However, these grants showed that an emphasis on evidence is both possible and 


popular, and they provided significant funding to grow many evidence-backed programs. 


i3’s and EIR’s successes should not solely be evaluated on how many of the grants 
worked and received positive evaluation results but also on how those successful grant- 
ees affect the field by growing directly or indirectly. The United States needs to increase 
investments in educational research and development, and the i3/EIR model can con- 


tinue to play a part in identifying, evaluating, and scaling promising innovations. 


Neil Campbell is the director of innovation for K-12 Education Policy at the Center for 
American Progress. Abby Quirk is a research associate for K-12 Education at the Center. 


This issue brief is part of the Moonshot for Kids project, a joint initiative from the Center for 
American Progress and the Thomas B. Fordham Institute to explore the rationale, potential, 
and possible design of a sizable new investment—whether by the federal government or 
large-scale philanthropy—in basic and applied research and development that leads to inno- 
vation on behalf of America’s children.” 
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